Papers with Preference Optimization
Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process (2025.acl-long)
Copied to clipboard
| Challenge: | Supervised Fine-Tuning (SFT) and Preference Optimization (PO) are key processes for aligning Language Models with human preferences post pre-training. |
| Approach: | They propose to combine Supervised Fine-Tuning and Preference Optimization (PO) with two sub-processes defined at token level within the Markov Decision Process (MDP) |
| Outcome: | The proposed process performs comparably or even superiorly to SFT and some typical PO methods across several tasks, particularly those requires generation, reasoning, and fact-following abilities. |
A Dual-Phase Self-Evolution Framework for Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing strategies to optimize LLMs through pretraining fail to enhance domain cognition. |
| Approach: | They propose a dual-phase self-evolution framework that integrates user preference adaptation and domain-specific competence to optimize LLMs. |
| Outcome: | The proposed framework outperforms Supervised Fine-Tuning, Preference Optimization, and Memory-Augmented baselines on general NLP benchmarks and long-term dialogue tasks. |
AlignCap: Aligning Speech Emotion Captioning to Human Preferences (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for speech emotion capture often produce hallucinations and lose generalization on unseen speech. |
| Approach: | They propose to align speech emotion captioning to human preference based on large language model (LLM) and human preference regularization to eliminate factuality and faithfulness hallucinations. |
| Outcome: | Experiments show that AlignCap performs better than existing methods on Zero-shot SEC task. |
SPO: Self Preference Optimization with Self Regularization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing reference-free preference optimization methods exhibit higher training efficiency but are prone to overoptimization, leading to performance degradation. |
| Approach: | They propose a reference-free preference optimization method that replaces the logsigmoid loss function with a SiLU function to improve the model's performance. |
| Outcome: | The proposed method achieves 7% improvement over SimPO on AlpacaEval 2 and MT-Bench. |
Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithms (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks, including instruction following, mathematical problem solving, and coding generation. |
| Approach: | They propose a method that truncates both preferred and dispreferred responses to match the shorter one’s length. |
| Outcome: | The proposed approach improves over standard implementations and achieves 11.8 points in AlpacaEval 2 and overall improvements across downstream tasks. |
CoTD-PO: Chain-of-Thought Distillation with Preference Optimization (2025.findings-emnlp)
Copied to clipboard
Lujie Niu, Haochen Sun, Fangkun Zhao, Sheng Chen, Zimeng Bai, Jiawei Zhang, Caixia Yuan, Xiaojie Wang
| Challenge: | Existing methods for chain-of-thought distillation suffer from a distribution mismatch between teacher-generated training trajectories and the student model's own generative distribution. |
| Approach: | They propose a framework that shifts the training paradigm from passive imitation to active trajectory exploration by allowing students to sample their own answer paths. |
| Outcome: | The proposed method outperforms standard CoT distillation baselines while mitigating mode collapse and preserving semantic diversity. |
Counterspeech the ultimate shield! Multi-Conditioned Counterspeech Generation through Attributed Prefix Learning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods to generate counterspeech based on intents are limited to single attributed . however, a holistic approach that considers multiple attributes simultaneously yields more nuanced and effective responses. |
| Approach: | They propose a framework that leverages hierarchical prefix learning with preference optimization to generate more constructive counterspeech. |
| Outcome: | The proposed framework improves intent conformity and emotion labels in 13,973 counterspeech instances. |
HARPO: Hierarchical Agentic Reasoning for User-Aligned Conversational Recommendation (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have significantly improved conversational recommender systems performance. |
| Approach: | They propose a framework that reframes conversational recommendation as a structured decision-making process optimized for multi-dimensional recommendation quality. |
| Outcome: | The proposed framework improves on ReDial, INSPIRED, and MUSE while maintaining competitive response quality. |